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14. ABSTRACT 


The  goal  of  this  research  is  to  determine  how  patient  safety  and  the  avoidance  of  medical  error  can  be  effectively  taught  to  student  nurses  in 
a  simulated  setting.  A  crossover  experiment  will  expose  subjects  to  a)  standardized  patients,  b)  high-fidelity,  and  c)  virtual  simulations,  to 
teach  specific  patient  safety  strategies.  Each  single  exposure  will  result  in  less  than  a  75%  mastery  of  the  critical  elements  of  the  exercise. 
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INTRODUCTION 


In  current  healthcare  settings,  the  opportunity  to  learn  is  restricted  by:  a)  the  availability  of  clinical  learning  opportunity  b)  the 
competition  among  different  levels  of  education  (doctors,  nurses,  paramedics)  to  learn  in  situ  and  c)  the  financial  and  emotional  cost 
(both  to  patient  and  health  care  providers)  of  medical  error  on  the  part  of  novice  learners.  Patient  safety  concerns  have  shaped  the  way 
in  which  students  can  interact  with  patients  at  the  bedside,  and  the  way  in  which  educators  now  view  the  “clinical  learning  lab”.  Ziv  et 
al 3  take  the  position  that  the  use  of  simulation,  wherever  feasible,  conveys  a  critical  educational  and  ethical  message  to  all:  patients 
are  to  be  protected  whenever  possible  and  they  are  not  commodities  to  be  used  as  conveniences  of  training. 


BODY 

STUDY  DESIGN 

The  design  is  a  quasi-experimental,  crossover  study.  Crossover  studies  have  wide  use  in  both  healthcare  and  educational  settings  and 
are  assistive  in  defining  which,  if  any,  treatments  have  greater  utility  over  the  others.  In  crossover  studies,  each  subject  serves  as 
his/her  own  control. 

The  experiment  focuses  on  two  aspects  of  simulation:  assessment  and  training.  Assessment  defines  to  what  extent  simulation  alone 
provides  a  learning  experience  that  can  be  measured  on  a  cognitive  exam  conducted  following  the  simulation  and  assesses  the 
psychomotor,  cognitive  and  affective  levels  of  ability  of  students.  Training  simulation  refers  to  what  extent  the  three  modalities 
(virtual,  standardized  patients  and  simulated  learning  experience)  affect  scores  on  both  a  post-test  simulation  and  a  post-test  cognitive 
exam. 

Students  enrolled  in  the  baccalaureate  Nursing  program  at  the  University  of  San  Francisco  were  recruited.  Case-scenarios  were  used  in 
all  three  exposures  (standardized  patient,  virtual  and  high-fidelity/simulated  learning  experience)  and  were  written  by  experts  and 
validated  using  content  validity  by  three  independent  faculty  experts.  The  standardized  patient  (SP)  exposure  consisted  of  theater 
student  majors  at  USF  “acting”  out  the  scenarios.  The  “virtual”  (VP)  exposure  consisted  of  a  series  of  video  scenarios  played  out  by 
faculty  that  present  the  subject  with  computer-based  recognition  and  response  choices.  The  "high-fidelity”  or  “simulated  learning 
experience”  (SLE)  consisted  of  scenarios  presented  in  a  manikin-based  format.  Students  were  randomized  to  all  three  experiments  in 
differing  order. 

Error  was  embedded  in  the  case  scenario  but  the  study  objective  (recognition  of  error)  will  be  unknown  to  the  student.  The  scenarios 
will  be  designed  so  that  the  student  expects  to  address  a  different  outcome  or  problem;  tune  to  recognition  and  response  will  be  a 
factor  when  considering  “mastery”.  Mastery  is  defined  as  meeting  an  acceptable  passing  score  (75%)  that  is  consistent  with  the 
criterion  used  to  assess  students  in  this  nursing  program. 

Using  statistical  analysis,  the  incremental  effect  of  each  treatment  will  be  determined  by  comparing  students’  scores  on  cognitive 
exams  before  and  after  the  treatment.  Since  the  ordering  of  the  treatments  may  also  have  an  effect  on  learning  and  retention,  a  broader 
analysis  will  be  conducted  to  evaluate  the  impact  of  not  only  the  treatment,  but  also  the  timing  of  the  treatment.  There  are  two  factors 
that  should  be  considered  here:  First,  a  treatment  may  have  a  different  effect  if  it  is  administered  first,  second  or  third.  Second,  there 
may  be  interactions  among  treatments,  where,  for  example,  administering  Treatment  A  then  Treatment  B  is  significantly  more 
effective  than  administering  Treatment  C  and  then  Treatment  B.  These  interactions  will  also  be  analyzed. 

KEY  RESEARCH  ACCOMPLISHMENTS 

RESEARCH  TO  DATE:  FIRST  YEAR  (2011-2012) 

In  order  to  complete  the  research,  three  (3)  existing  classrooms  were  renovated  to  create  a  research  laboratory 
for  the  simulation  study  design.  To  wit:  two  (2)  high-fidelity  rooms,  equipped  to  mimic  hospital  rooms,  two 
standardized  patient  rooms  equipped  to  mimic  physician  offices  and  one  (1)  computed  lab  were  renovated.  All 
of  the  rooms  were  soundproofed  to  avoid  potential  contamination  among  subjects  and  corridors  were  created  to 
avoid  subjects  meeting  in  hallways.  Because  this  is  a  crossover  study  in  which  all  subjects  will  undergo  each  of 
the  three  experiments,  particular  attention  was  paid  to  the  potential  for  subjects  sharing  information. 

Renovation  of  the  lab  was  not  complete  until  May,  2012.  The  first  subject  were  enrolled  in  June,  2012. 

This  first  study  served  as  a  pilot  test  to  determine: 

•  time  needed  for  pre/post  testing 

•  time  needed  for  each  experiment 

•  flow  of  subjects  through  lab 


Respondents 


Over  the  Summer  2012  term,  20  students  were  approached  about  participating  in  the  simulation  study.  While  there  is  no  statistically 
significant  difference  in  the  mean  GPAs  between  those  agreeing  and  those  not,  the  variability  in  the  GPAs  is  greater.  Both  the  largest 
(3.99)  and  the  smallest  (2.78)  GPAs  were  among  the  non-participants.  Table  1  summarizes  the  GPA  statistics. 


Table  1:  GPA  statistics  for  study  participants  and  non-participants 


Number 

Average  GPA 

Standard  Deviation  of  GPA 

Participants 

15 

3.488 

0.2737 

Non-Participants 

5 

3.494 

0.4565 

Of  the  15  participants,  3  never  came  to  do  the  study.  Their  average  GPA  is  lower  than  that  of  those  who  came  for  at  least  one 
treatment  (3.22  vs.  3.55). 


TESTING  THE  RESEARCH  OBJECTIVES 

The  following  objectives  were  tested  in  the  pilot  study  Summer  2012: 

Research  Objective  #1:  To  determine  if  one  of  three  methods  or  a  combination  of  simulation  instruction  provides  greater 
mastery  in  patient  safety  instruction  for  pre-licensure  students. 

Research  Objective  #2:  To  determine  if  the  sequence  of  three  methods  of  simulation  instruction  predicts  mastery  of  patient 
safety  instruction  for  pre-licensure  students. 

Research  Objective  #3:  To  determine  if  prior  exposure  during  training  simulations  can  predict  how  students  will  perform 
relative  to  non-participants  on  a  final  simulation. 


STATISTICAL  ANALYSIS 
Exam  Question  Evaluation 

The  10-question  pre-  and  post-test  exam  is  evaluated  for  the  effectiveness  of  the  questions.  Table  2  shows  the  difficulty  level  of  the 
questions.  Difficulty  level  is  measured  as  a  percentage  of  students  answering  the  question  correctly;  observations  from  all  pre-  (post- 
)exams  are  aggregated. 


Table  2:  Percentage  of  students  answering  questions  correctly  before  and  after  treatment 


Question 

Pre-test  Percentage 

Post-test  Percentage 

1 

85.71% 

90.48% 

2 

76.19% 

76.19% 

3 

23.81% 

23.81% 

4 

95.24% 

100.00% 

5 

76.19% 

85.71% 

6 

57.14% 

57.14% 

7 

30.43% 

34.78% 

8 

90.48% 

85.71% 

9 

42.86% 

42.86% 

10 

47.62% 

52.38% 

Based  on  the  difficulty  level,  question  4  should  be  eliminated  because  too  many  students  answer  correctly.  Question  8  may  also  be 
considered  for  elimination.  Question  3  may  be  too  challenging  for  students;  in  the  second  and  third  experiments,  0%  and  17%  of 
students  answered  the  question  correctly,  respectively. 


Simulation  Performance 


Participants  are  scored  on  a  1 3-item  checklist  for  desired  actions  during  patient  interactions.  Scores  are  expressed  as  the  percentage  of 
items  performed.  Students  are  randomly  assigned  to  one  of  four  groups.  Each  group  is  exposed  to  simulation  treatments  in  a  different 
order,  see  Table  3: 


Table  3:  Order  of  simulation  exposure  per  group 


Group 

1st  Treatment 

lUAB^ 

3rd  Treatment 

A  (control  group) 

High  fidelity 

B 

Standardized  patient 

Virtual  reality 

High  fidelity 

C 

Virtual  reality 

High  fidelity 

Standardized  patient 

D 

High  fidelity 

Standardized  patient 

Virtual  reality 

Table  4  shows  participant  scores  for  each  of  the  treatments,  as  well  as  the  sample  size.  Because  of  the  extremely  small  sample  sizes, 
no  general  conclusions  can  be  drawn  about  changes  in  performance. 


Table  4:  Average  participant  score  (sample  size) 


Group 

2nd  Treatment 

A  (control  group) 

61.54%  (2) 

B 

- 

53.85%  (1) 

C 

53.85%  (3) 

D 

41.03%  (3) 

38.46%  (2) 

80%  (2) 

Table  5  shows  the  average  score  across  groups  for  each  of  the  types  of  simulation.  There  appears  to  be  no  significant  difference 
between  performance  on  a  standardized  patient  and  a  high  fidelity  environment. 


Table  5:  Average  scores  for  each  treatment  type 


Treatment 

Average  Score 

Standardized  patient 

51.65% 

Virtual  reality 

70% 

High  fidelity 

49.04% 

Because  of  the  extremely  small  sample  size,  no  real  conclusions  can  be  drawn  from  the  pre-  and  post-tests  conducted  on  participants. 
Table  6  shows  the  percent  change  in  exam  score  (post-test  %  correct  -  pre-test  %  correct),  the  number  of  observations  with  no  change 
in  exam  score,  and  the  number  of  observations.  Treatment  3  has  the  largest  change,  with  2  participants  each  increasing  their  score  by 
40%. 

Participant  Survey 

After  all  3  treatments,  participants  were  asked  to  complete  a  25-question  survey  about  their  experience.  Each  question  was  answered 
by  4  or  5  participants.  20  of  the  questions  asked  specifically  about  the  students’  perception  of  the  simulations  and  the  learning  effect, 
e.g.,  “I  feel  more  confident  recognizing  changes  in  the  patient’s  condition.”  The  other  5  questions  (#1 1,  13,  14, 23, 25)  may  be 
interpreted  more  generally,  e.g.,  “I  am  having  difficulty  prioritizing  patient  care  needs.”  Table  7  summarizes  the  number  of  students 
feeling  negatively,  neutrally,  positively,  and  very  positively  in  both  categories. 


Table  7:  Tabulation  of  participation  perceptions 


Very  positive 

Positive 

Neutral 

Simulation- specific  question 

6 

9 

4 

1 

General  question 

_ 5 _ 

The  only  question  that  received  a  negative  response  was  the  first  question,  “I  feel  better  prepared  to  care  for  real  patients.”  The  vast 
majority  of  all  other  questions  were  answered  positively. 


RESEARCH  OBJECTIVE  NOT  YET  TESTED 


Research  Objective  tt4:  To  design  a  rubric/model  for  safety  instruction  capable  of  stratifying  average  and  exceptional 
performance. 


The  academic  year  2012-2013  will  be  used  to  enroll  two  (2)  additional  cohorts  of  subjects.  Development  of  tiie  rubric  will  begin 
Summer,  2013. 


CONCLUSION 

Enrollment  and  retention  of  subjects  is  a  significant  issue  hi  this  longitudinal  study  that  requires  three  (3)  visits  to  the  laboratory  over 
one  (I)  semester.  Strategies  will  need  to  be  identified  to  overcome  this  issue. 
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